Adaptive Auto Exposure and Dynamic Range Compensation

ABSTRACT

This disclosure pertains to systems, methods, and computer readable media for extending the dynamic range of images using an operation referred to herein as “Adaptive Auto Exposure” (AAE). According to the embodiments disclosed herein, the AAE-enabled higher dynamic range capture operations are accomplished without blending multiple or bracketed exposure captures (as is the case with traditional high dynamic range (HDR) photography). AAE also enables high signal-to-noise ratio (SNR) rendering when scene content allows for it and/or certain highlight clipping is tolerable. Decisions with regard to preferred AE strategies may be based, at least in part, on one or more of the following: sensor characteristics; scene content; and pre-defined preferences under different scenarios.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims priority to U.S. Provisional Patent ApplicationNo. 61/875,657, entitled, “Adaptive Auto Exposure and Dynamic RangeCompression with Noise Balancing using Lens Shading Modulation” andfiled Sep. 9, 2013 (“the '657 application”). The '657 application isalso hereby incorporated by reference in its entirety.

BACKGROUND

This disclosure relates generally to the field of image processing and,more particularly, to various techniques for use in adaptively autoexposing and improving the signal-to-noise ratio of digital imagescaptured by an image sensor, e.g., in a personal electronic device.

Today, many personal electronic devices come equipped with digitalcameras. Often, these devices perform many functions, and, as aconsequence, the digital image sensors included in these devices mustoften be smaller than the sensors in conventional cameras. Further, thecamera hardware in these devices often have smaller dynamic ranges andlack sophisticated features sometimes found in larger,professional-style conventional cameras such as manual exposure controlsand manual focus. Thus, it is important that digital cameras in personalelectronic devices be able to produce visually appealing images in awide variety of lighting and scene situations with limited or nointeraction from the user, as well as in a computationally andcost-effective manner.

One feature that has been implemented in some digital cameras tocompensate for lack of dynamic range and create visually appealingimages is known as “auto exposure.” Auto exposure (AE) can be definedgenerally as any algorithm that automatically calculates and/ormanipulates certain camera exposure parameters, e.g., exposure time,ISO, or f-number, in such a way that the currently exposed scene iscaptured in a desirable manner.

Auto exposure algorithms are often employed in conjunction with imagesensors having small dynamic ranges because the dynamic range of lightin a given scene, i.e., from absolute darkness to bright sunlight, ismuch larger than the range of light that some image sensors—such asthose often found in personal electronic devices—are capable ofcapturing. However, the exposure parameter adjustments, e.g., gainadjustments, implemented by traditional AE algorithms may cause thesignal to be “clipped,” that is, be pushed to a value above (or below)what the camera's hardware is capable of storing. This clipping processcan result in the loss of valuable image detail.

In addition to the above-mentioned clipping issue, image noise isanother challenge the image processing pipeline needs to deal with. Theuse of longer sensor integration time is one technique that may beemployed to attempt to enhance the image signal while reducing therandom noise. Often, however, image processing pipelines do not fullyutilize this image sensor behavior when the scene to be captured allowsfor longer integration time before signal clipping occurs.

The inventors have realized new and non-obvious ways to achieve lowernoise image captures and process the resulting captured images withoutclipping the image signal. The inventors have also realized new andnon-obvious ways to perform lens shading correction (LSC) operationsthat modulate gains based on scene lux level and lens focus distance.

SUMMARY

Imaging sensors have limited capability to hold electrons generated fromintegrating incoming light energy over exposure time. When moreelectrons are produced through exposure to light than what an imagingsensor pixel can hold, image signal clipping will occur. Before clippingoccurs, longer integration times may be employed to produce a less noisyand higher quality image. The ratio of the highest non-clipped pixelvalue to the lowest non-clipped pixel value may be used to define the“dynamic range” of the captured image. The role of traditional AE is tofind a balanced exposure time, whereby the scene is well exposed—withoutcausing significant signal clipping—in order to achieve a goodcompromise between noise and dynamic range. The limiting factor of thisstrategy in brightly-lit scene conditions is signal clipping. Thelimiting factor of this strategy under dimly-lit scene conditions isimage noise.

Raw imaging sensor data needs to be processed by image processinghardware and software to be useable for viewing. Often, the digitalprocessing is employed to improve the image quality of the raw sensordata, such as filtering out the noise, correcting the color, and evenenhancing the image to achieve pleasing rendering quality. Over the pastyears, the research community has come up with more and more powerfulnoise filtering techniques to effectively remove the sensor noise. Moreadvanced sensor data noise filtering allows a noisier sensor input toresult in visually pleasing images in a greater amount of capturedimages. This opens the opportunity to use shorter exposure times, whilestill producing a quality final output image. One benefit of shorterexposure time is to allow more high luminance (i.e., bright) scenecontent to be properly exposed without causing signal clipping. Thisprovides the possibility to produce higher dynamic range image contentusing the same image sensor but a different exposure strategy and imageprocessing algorithms.

As noise filtering algorithms advance, they provide some leverage toallow the AE exposure strategies to be more flexible. However, noisefiltering is often not powerful enough to make higher quality resultingimages, as compared to simply using a longer exposure time to accumulatemore electrons to physically have a better (i.e., quieter) image sensorinput to start. In addition, digital noise filtering often has theundesirable side effect of suppressing image detail and low contrasttextures, thereby preventing a faithful rendering of the scene'scontent. This is another difficulty that AE systems attempt to dealwith. One commonly-used AE target value in the industry is about 20% ofthe full capacity of the sensor. This corresponds to the 18% neutralsurface exposure target in the film photography era. With a simplifiedimage sensor noise behavior formula, the signal-to-noise ratio of thesensor pixel value may be set to the square root of the signal itself.With this formula, the more electrons the sensor pixel collects, thehigher the signal-to-noise ratio of the image signal will become. So, toaccount for increased signal noise, one approach is to increase theintegration time as much as possible before signal clipping occurs. Whenthe signal is weak, increasing integration time is often a naturalchoice to enhance image quality. It is the bright—yet, relatively lowcontrast—scenes that provide the opportunity to have a bettersignal-to-noise ratio with a more flexible AE strategy. Thus, in oneembodiment described herein, the 20% full capacity target valueintegration strategy can be altered to boost the signal—with less noiseand no clipping.

In summary, the disclosure relates to scene-adaptive AE strategies anddigital signal processing techniques that may be employed to achieve adesired image capture and rendering experience according to the imagesensor's characteristics and rendering purpose. In the followingdescription of the disclosure, “EV0” refers to the standard exposurestrategy. “EV+” refers to an exposure target that is larger than the EV0target. “EV−” refers to an exposure target that is smaller than the EV0target. The EV0 exposure strategy may be different from manufacturer tomanufacturer of image sensors.

This disclosure also relates to strategies and techniques for performinglens shading correction operations that modulate gains based on scenelux level and lens focus distance. These gains compensate for both colorlens shading (i.e., the deviation between R, G, and B channels) andvignetting (i.e., the drop off in pixel intensity around the edges of animage). As scene illuminance increases, the sensor captures more signalfrom the actual scene, and the lens shading effects begin to appear. Todeal with the situation, the lens shading gains are configured toadaptively ‘scale down’ when scene lux approaches zero and ‘scale up’when scene lux changes from near zero to become larger. The lens shadinggain may also be modulated based on the focus distance. For opticalsystems without zoom, the inventors have discovered that the amount oflens shading fall off changes as focus distance changes.

Thus, in one embodiment disclosed herein, a non-transitory programstorage device, readable by a programmable control device, may compriseinstructions stored thereon to cause the programmable control device to:obtain a first AE target for a first image of a scene to be captured byan image sensor; obtain the first image captured based, at least inpart, on the first AE target, wherein the first image comprises a firstplurality of pixels; determine whether there is unused dynamic range inthe captured first image; determine a second AE target different fromthe first AE target if it is determined that there is unused dynamicrange; obtain a second image captured based, at least in part, on thesecond AE target, wherein the second image comprises a second pluralityof pixels; generate one or more Dynamic Range Compensation (DRC) curvesfor the second image based, at least in part, on the first AE target;and perform tone mapping on the second plurality of pixels using thegenerated one or more DRC curves, wherein the tone mapping performed onthe second plurality of pixels is configured to render the second imageto match a rendering as if the second image was captured with the firstAE target. A global DRC curve (or a plurality of spatially-varying localDRC curves) may be used to digitally gain down the low and middle tonesignal of EV+ image to maintain the luminance of EV0. The DRC curve(s)may also be used to expand the contrast on the highlights of the EV+image while ensuring that white pixels (i.e., those pixels having themaximum possible luminance value) remain white to avoid“under-clamping.” The term “under-clamping” refers to the process ofpulling down the white over-exposed image pixels to a gray tone. Thismay lead to an undesired appearance of image highlights. In analogy tothe human visual system, very bright scene highlights are expected to beperceived as pure white, but not gray, and thus the image processingpipeline, according to some embodiments, may desirably pertain whiteover-exposed pixels, rather than graying them out.

In another embodiment disclosed herein, a non-transitory program storagedevice, readable by a programmable control device, may compriseinstructions stored thereon to cause the programmable control device to:obtain a first AE target for a first image of a scene to be captured byan image sensor; obtain the first image captured based, at least inpart, on the first AE target, wherein the first image comprises a firstplurality of pixels; determine whether any of the first plurality ofpixels comprise clipped highlight pixels; determine a second AE targetdifferent from the first AE target if it is determined that clippedhighlight pixels are present; obtain a second image captured based, atleast in part, on the second AE target, wherein the second imagecomprises a second plurality of pixels; generate one or more DRC curvesfor the second image based, at least in part, on the first AE target;and perform tone mapping on the second plurality of pixels using thegenerated one or more DRC curves, wherein the tone mapping performed onthe second plurality of pixels is configured to render the second imageto match a rendering luminance as if the second image was captured withthe first AE target. A global DRC curve (or a plurality ofspatially-varying local DRC curves) may be used to digitally gain up thelowlight and mid-tone regions of the image signal to maintain theluminance of EV0. The DRC curve(s) may also be used to compress thehighlight regions of the EV− image so that rendering does not causesignal clipping from the linear gain up operation.

In another embodiment disclosed herein, a non-transitory program storagedevice, readable by a programmable control device, may compriseinstructions stored thereon to cause the programmable control device to:obtain a first image of a scene, wherein the first image comprises afirst plurality of pixels; determine a scene illuminance level for thefirst image; determine a lens shading correction level based, at leastin part, on the determined scene illuminance level; and modulate thedetermined lens shading correction level based, at least in part, on thedetermined scene illuminance level, wherein the determined lens shadingcorrection level encompasses both color shading gain and vignettinggain.

In another embodiment disclosed herein, a non-transitory program storagedevice, readable by a programmable control device, may compriseinstructions stored thereon to cause the programmable control device to:obtain a first image of a scene, wherein the first image comprises afirst plurality of pixels; obtain a focal distance of a lens used tocapture the first image; determine a lens shading correction levelbased, at least in part, on the obtained focal distance; and modulatethe determined lens shading correction level based, at least in part, onthe obtained focal distance, wherein the determined lens shadingcorrection level encompasses both color shading gain and vignettinggain.

In still other embodiments, the techniques described herein may beimplemented in apparatuses and/or systems, such as electronic deviceshaving image capture capabilities, memory, and programmable controldevices.

BRIEF DESCRIPTION OF THE DRAWINGS

The patent or application file contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawing(s) will be provided by the Office upon request and paymentof the necessary fee.

FIG. 1 shows an adaptive AE and tone mapping overview in block diagramform, according to one embodiment.

FIG. 2 shows a hue-preserving RGB tone mapping process in block diagramform, according to one embodiment.

FIG. 3 illustrates a typical DRC curve in graph form, according to oneembodiment.

FIG. 4A shows an EV+ adaptive AE and tone mapping overview in flowchartform, according to one embodiment.

FIG. 4B illustrates an EV+ adaptive AE scenario in line graph andhistogram forms.

FIG. 5A shows an EV− adaptive AE and tone mapping overview in flowchartform, according to another embodiment.

FIG. 5B illustrates an EV− adaptive AE scenario in line graph andhistogram forms.

FIG. 6A shows an adaptive AE process with combined color shading andvignetting correction in block diagram form, according to oneembodiment.

FIG. 6B shows an adaptive AE process with decoupled color shading andvignetting correction in block diagram form, according to oneembodiment.

FIG. 7 illustrates LSC gain curves before and after normalization ingraph form, according to one embodiment.

FIG. 8A illustrates low-light vignetting gain modulation curves in graphform, according to one embodiment.

FIG. 8B illustrates focus position-dependent vignetting gain modulationcurves in graph form, according to one embodiment.

FIG. 9 shows, in block diagram form, an illustrative electronic devicethat may implement one or more of the described adaptive AE and tonemapping image processing operations.

DETAILED DESCRIPTION

This disclosure pertains to systems, methods, and computer readablemedia for extending the dynamic range of images using an operationreferred to herein as “Adaptive Auto Exposure” (AAE). According to theembodiments disclosed herein, the AAE-enabled higher dynamic rangecapture operations are accomplished without taking multiple or bracketedexposure captures (as is the case with traditional high dynamic range(HDR) photography). AAE also enables high signal-to-noise ratio (SNR)rendering when the scene content has relatively low dynamic range and/orcertain highlight clipping is tolerable. Decisions with regard topreferred AE strategies may be based, at least in part, on one or moreof the following: sensor characteristics; scene content; and pre-definedpreferences under different scenarios.

In the generation of high dynamic range images prior to the use of thedynamic range extension techniques described herein, the user of thecamera had to either take multiple images bracketed at differentexposure levels and attempt to fuse the images together—which oftenresulted in blurriness, ghosting, or other unwanted artifacts associatedwith movement of the camera and/or objects within the image betweensuccessive exposures. With the use of the AAE techniques describedherein, however, the user of the imaging system may simultaneouslycapture an extended dynamic range image, as well as avoid the need tofuse together multiple images taken at different bracketed exposurelevels. The “EV−” exposure setting described above (wherein an exposuretarget is set that is smaller than the “EV0” target) allows more scenecontent be properly captured by the sensor. The “up processing” thatfollows the use of an “EV−” exposure setting subsequently tone maps thehigher dynamic range content of the image to a regular dynamic rangeimage—with the rendering result appearing close to an “EV0” renderingresult but with additional highlight benefits which would become clippedin the “EV0” image otherwise. For the “EV−” case, the amount of “upprocessing” that may be done to bring in more highlights to theprocessed image is limited by sensor noise. The “EV+” exposure settingdescribed above (wherein an exposure target is set that is higher thanthe “EV0” target) improves the signal-to-noise ratio of the capturedimage compared to what is acquired by the “EV0” exposure setting. The“down processing” that follows the use of an “EV+” exposure settingsubsequently maps the lower dynamic range image digitally to match theappearance of “EV0” rendering. For the “EV+” case, the amount of “downprocessing” that may be done is limited mainly by the amount of motionblur in the captured image. For example, the upper bound of the EV+ maybe determined based, at least in part, on the system's capability tocontrol motion blur. The amount of motion blur may be determined inreal-time or near real-time and used as a limiter on the amount of “downprocessing” that is performed on the image.

In the following description, for purposes of explanation, numerousspecific details are set forth in order to provide a thoroughunderstanding of the inventive concept. As part of this description,some of this disclosure's drawings represent structures and devices inblock diagram form in order to avoid obscuring the invention. In theinterest of clarity, not all features of an actual implementation aredescribed in this specification. Moreover, the language used in thisdisclosure has been principally selected for readability andinstructional purposes, and may not have been selected to delineate orcircumscribe the inventive subject matter, resort to the claims beingnecessary to determine such inventive subject matter. Reference in thisdisclosure to “one embodiment” or to “an embodiment” means that aparticular feature, structure, or characteristic described in connectionwith the embodiment is included in at least one implementation of theinvention, and multiple references to “one embodiment” or “anembodiment” should not be understood as necessarily all referring to thesame embodiment.

It will be appreciated that, in the development of any actualimplementation (as in any development project), numerous decisions mustbe made to achieve the developers' specific goals (e.g., compliance withsystem- and business-related constraints), and that these goals may varyfrom one implementation to another. It will also be appreciated thatsuch development efforts might be complex and time-consuming, but wouldnevertheless be a routine undertaking for those of ordinary skill in thedesign of an implementation of image processing systems having thebenefit of this disclosure.

Referring now to FIG. 1, an overview diagram 100 is shown. The overviewdiagram is composed of two primary parts, hardware block 105 andfirmware block 110. Hardware block 105 may, for example, comprise aprogrammable microprocessor such as the ARM® Cortex A7, Apple A6 systemon a chip (SoC), or other suitable processor, as well as an image sensorpackage, such as a CMOS sensor. Firmware block 110 may comprise acontrol system, e.g., process and timing control block 180, which mayalso be used to set up and program the hardware block 105 on aframe-by-frame basis via pathway 185, e.g., based on various imagestatistics related to the image processing pathway gathered by thehardware block 105. Hardware block 105 may comprise multiple functionalblocks, including an Image Statistics block 115, a lens shadingcorrection block 120, an RGB interpolation block 140, a local/globaltone mapping block 125, and gamma correction block 130.

Likewise, firmware block 110 may comprise multiple functional blocks,including an Auto function block 135 (which may include camera functionssuch as Auto White Balance (AWB), Auto Exposure (AE), and Auto Focus(AF)) and a local/global tone curve calculation block 145 to compressthe higher dynamic content to a normal range, each of which will beexplained in further detail below. In the case of local, i.e.,spatially-varying tone curve calculation, DRC curves may be determinedat, e.g., individual pixel locations or at a plurality of locationsthroughout the image. Various image statistics are shared betweendifferent functional blocks of firmware block 110 and hardware block105, e.g., via image statistics transfer interface 175. For example,thumbnails and histograms may be sent from Image Statistics block 115 toAuto function block 135.

Hue-Preserving RGB Tone Mapping Operation

The hue-preserving RGB tone mapping operation, as described herein, maycomprise a global or local tone modification process. A block diagram200 illustrating the tone mapping process, according to one embodiment,is shown in FIG. 2. First, luminance values are calculated from the RGBsgoing into the tone mapping block 205. For instance, the luminance valuemay be a linear combination of R, G, and B, or the maximum of R, G, andB, or a mixture of both aforementioned definitions. Next, gains arecomputed 210 from a global/local tone mapping curve, and the gains areapplied to the input RGB values to generate the desired output RGBlevels. The tone mapping operation 200 may scale the RGB values by again, g=Y_(out)/Y_(in), where Y_(in) and Y_(out) are luminancerepresentations of the input and output to block 210, respectively. Incontrast to global tone mapping as is commonly applied in the gammacorrection portion of the image processing pipeline, for instance,hue-preserving RGB tone modification does not shift the hues of RGBssince the input to output mapping is not applied to RGB channelsseparately, but to a common luminance channel. Thus, all RGB values arescaled with the same local gain, g.

Dynamic Range Compensation (DRC) Curves

Dynamic Range Compensation (DRC) curves, as described herein, provide asimple, yet effective, novel approach for tone compensation, e.g., thecompression and/or expansion of high dynamic range content into thenormal dynamic range for encoding and viewing. The DRC curves may beconstructed such that they begin with a first segment having a linearslope followed by a non-linear (typically compressing) second segment.The linear segment may be used to gain up the mid and darker tones'linearly, in order to match the EV0 exposure image's luminance. Thesecond (i.e., non-linear) segment may be used to compress highlighttones, e.g., due to the additional dynamic range resulting from EV−exposure. The partition of the linear segment and nonlinear segmentdepends on the EV− value and image content statistics.

DRC Curve Construction

For the DRC curve construction, a piece-wise linear function with N+1equidistant samples per curve may be used. In some embodiments, N may bechosen to be 32. Let f_(DRC)(n) describe the N+1 samples of thepiece-wise linear curve where n is the sample number. A reasonableconstraint is imposed, which is to preserve black and white after themapping, i.e., f_(DRC)(0)=0 and f_(DRC)(N)=1.

The following exemplary formula uses negative EV bias, EV−, thatrequires an “up gaining” of the image signal to restore the EV0 imageappearance. This gain is denoted as g_(EV) in the description herein.

Based on the EV gain, g_(EV), a knee value n_(knee)ε

, 0<n_(knee)<N is obtained that separates a linear segment from anon-linear segment in the function, f_(DRC)(n):

$\begin{matrix}{{n_{knee} = {n_{l\; i\; n}\left( {y_{{ma}\; x} - {\frac{y_{{ma}\; x} - y_{m\; i\; n}}{N}\left( {N + 1 - n_{{li}\; n}} \right)}} \right)}},} & \left( {{Eqn}.\mspace{14mu} 2} \right)\end{matrix}$

where n_(lin)=N/g_(EV), and the minimum and maximum knee value isspecified by y_(min) and y_(max) (e.g., y_(min)=0.7 and y_(max)=0.8).These values can be used for tuning to achieve a fair trade-off betweenhighlight compression and linearity in the low- and mid-tones.

The linear segment of the function f_(DRC)(n) may be defined asf_(lin)(n)=^(n)/_(N)g_(EV), and the final DRC curve may be definedrecursively as:

$\begin{matrix}{\mspace{20mu} {{f_{nonlin}(n)} = {s\left( {{f_{DRC}\left( {n - 1} \right)} + {p\left\lbrack {1 - {f_{DRC}\left( {n - 1} \right)}} \right\rbrack}} \right)}}} & \left( {{Eqn}.\mspace{14mu} 3} \right) \\{{f_{DRC}(n)} = \left\{ {\begin{matrix}{f_{l\; i\; n}(n)} & {{{if}\mspace{14mu} n} \leq \left\lfloor n_{knee} \right\rfloor} \\{{{wf}_{l\; i\; n}(n)} + {\left( {1 - w} \right){f_{nonlin}(n)}}} & {{{if}\mspace{14mu} n} = {\left\lfloor n_{knee} \right\rfloor + 1}} \\{f_{nonlin}(n)} & {otherwise}\end{matrix},} \right.} & \left( {{Eqn}.\mspace{14mu} 4} \right)\end{matrix}$

where the weight, w, smoothly blends the linear segment into thenon-linear segment and is assigned to the fractional part of n_(knee),i.e., w=mod (n_(knee), 1). The parameter, p, defines the shape of thecompression part of the curve, that is, the degree of non-linearity(banding). It can be parameterized with g_(EV) to have a differentcompression at different gain slopes. The parameter, s, is determinedfor a set of gain values, g_(EV), to fulfill the f_(DRC)(N)=1constraint. These values may be obtained using off-line optimization andmay be pre-calculated and stored in a lookup table (LUT). Note that thefinal DRC curves, f_(DRC)(n), can also be stored in a LUT for a range ofgain values, g_(EV), to achieve a better computational efficiency.

FIG. 3 illustrates an exemplary graph 300 of a DRC curve, f_(DRC)(n),with an EV gain of g_(EV)=1.977. The samples to the left of knee point305 represent the linear portion incorporating the EV gain, g_(EV). Thesamples to the right of knee point 305 represent the non-linear segmentof the curve, which aims to compress the dynamic range. The dot 305represents a weighted average of the linear and non-linear portions ofthe curve near the knee point.

AE Adaptation Strategy

When the input image statistics indicate that the scene does not haveenough range to fill up the sensor capacity, Adaptive AE (AAE) mayincrease the exposure target to consciously overexpose the image inorder to achieve a higher signal-to-noise level. An example of theabove-mentioned image statistics is an image histogram that provides theimage pixel luminance distribution for the image.

In contrast to the DRC tone mapping described above, which may beemployed in an underexposure situation, the dynamic range expansionprocess may be performed during tone mapping in an overexposuresituation. The basic task may be described as mapping the elevated pixelvalues down to the value that an “EV0” exposure would produce. Tomaintain white as white, the highlight portion of the tone expansioncurve will have gains larger than 1, which produces a highlight contraststretch effect. The HW/SW implementation of the dynamic range expansionprocess mirrors the DRC operation.

Positive-biased AE adaptation (“EV+”) depends first on the scenecontent. In one embodiment described herein, an image histogram may beused to decide if there is ‘unused’ sensor dynamic range available forthe particular scene to be captured in. If so, then the EV+ value can becalculated to achieve a predefined target dynamic filling rule. Oneexemplary rule comprises filling up the sensor dynamic range with thelower 99 percent of the total pixels based on pixel luminance. In otherwords, such an exemplary rule would allow 1% of the image content to beclipped. As the scene lux (i.e., illuminance level) gets lower, longerimaging sensor integration time is required to properly capture images.Longer integration times may result in blurry images, e.g., due tomotion of object within the scene, while the sensor is capturing light.This AE adaptation process may thus be stopped when the amount ofmotion-introduced blur is no longer acceptable.

Referring now to FIG. 4A, an adaptive AE and tone mapping overview foran EV+ scenario is shown in flowchart form 400, according to oneembodiment. First, a first image of a scene is obtained using a first AETarget (405). Then, the process determines that there is ‘unused’ sensordynamic range in the first image, i.e., that there are no pixels fallinginto the highest, say, n buckets of the image luminance histogram (410).Next, the process may obtain a second image of the scene using a secondAE Target, configured to utilize at least part of the ‘unused’ sensordynamic range (415). Next, the process may generate a DRC curve ormultiple curves (e.g., in the case of spatially-varying tone mapping)based, at least in part, on the second AE Target (420). Finally, tonemapping may be performed, with the DRC curve(s) being configured torender the image to match the rendering luminance as if the image wascaptured with the first AE Target, but with less signal noise than ifthe image had been captured at the first AE Target (425).

Referring now to FIG. 4B, an EV+ adaptive AE scenario is illustrated inboth line graph and histogram forms. Line graph 450 represents threedistinct lux zones over which the EV bias may vary in an EV+ scenario.First, in Zone 1, it is recognized that using a longer integration timewould make the image more prone to blurring, and thus the EV+ bias willbe disabled. Next, in Zone 2, the so-called “transition zone,” motionblur gets better as scene illuminance increases, so the EV+ bias may begradually increased. Finally, in Zone 3, EV+ bias goes to full strength.

Turning to EV0 histogram 460, an exemplary pixel luminance distributionis shown, with the dashed line representing the peak of the histogram,i.e., the luminance level at which the most pixels are located. Next,the EV+ histogram 470 is shown, representing the luminance distributionof pixels in the image that has been captured after the AE target hasbeen biased upward. As such, the peak of the histogram has moved alongthe horizontal axis closer to the “MAX” luminance level. Finally,post-process histogram 480 represents the distribution of pixelluminance values after the pixels have been rendered as if the image wascaptured at the EV0 exposure level. As shown in histogram 480, imagehistogram looks largely similar to the EV0 histogram depicted inhistogram 460, but with better noise reduction due to the increasedsensor integration time used to capture the EV+ biased image.

Negative-biased AE adaptation (“EV−” or “Dynamic Range extension”) maycomprise a more constrained adaptation process, as it will result inincreased image noise, as compared to the positive-biased AE adaptation.The sensor noise depends on optical system parameters and sensorcharacteristics. For example, larger lenses and larger pixel areasresult in better noise performance due to the physical efficiency of theabsorption of light energy. More powerful noise filtering software andhardware may provide an extra buffer to leverage for AE adaptation. EV−adaptation may be applied to captures with sufficient scene lux. Therange of the adaptation, in other words, how much AE negative bias canbe achieved is noise-bounded, and thus varies from imaging system toimaging system and on a given implementation's noise tolerance criteria.The desired dynamic range and noise balance for a particular scene canthus depend, at least in part, on one or more of: scene lux; sensornoise characteristics; and a noise filter's ability to mask the elevatednoise.

Referring now to FIG. 5A, an adaptive AE and tone mapping overview foran EV− scenario is shown in flowchart form 500, according to oneembodiment. First, a first image of a scene is obtained using a first AETarget (505). Then, the process determines that there is highlightclipping in the first image (510). Next, the process may obtain a secondimage of the scene using a second AE Target, wherein the second AETarget is chosen to achieve a predefined target dynamic filling rule(515). Next, the process may generate a DRC curve or multiple curves(e.g., in the case of spatially-varying tone mapping) based, at least inpart, on the second AE Target (520). Finally, tone mapping may beperformed, with the DRC curve(s) being configured to render the image tomatch the rendering luminance as if the image was captured with thefirst AE Target, but with less highlight signal clipping than if theimage had been captured at the first AE Target (525).

Referring now to FIG. 5B, an EV− adaptive AE scenario is illustrated inboth line graph and histogram forms. Line graph 550 represents threedistinct lux zones over which the EV bias may vary in an EV− scenario.First, in Zone 1, it is recognized that using a shorter integration timewould boost the noise in the image, and thus the EV− bias will bedisabled. Next, in Zone 2, the so-called “transition zone,” noisebehavior gets better as scene illuminance increases, so the EV− bias maybe gradually increased. Finally, in Zone 3, EV− bias goes to fullstrength. The EV− boundary may be set to meet a maximum tolerable noiserequirement.

Turning to EV0 histogram 560, exemplary highlight clipping is shown viathe large percentage of pixels having the “MAX” luminance level. Next,the EV− histogram 570 is shown, representing the luminance distributionof pixels in the image that has been captured after the AE target basbeen biased downward. Finally, post-process histogram 580 represents thedistribution of pixel luminance values after the pixels have beenrendered as if the image was captured at the EV0 exposure level. Asshown in histogram 580, the low and middle tones experience little to nochange, but the high tones are adjusted such that the pixels at thehighest luminance levels are no longer clipped as they were in histogram560.

Referring now to FIG. 6A, an adaptive AE process with combined colorshading and vignetting correction is shown in block diagram form 600,according to one embodiment. First, the camera focus position and scenelux level may be sent to vignetting modulation block 605. Once theamount of vignetting modulation has been determined, the color shadingand vignetting correction may be applied to the image input signal atblock 610. Simultaneously, the image input signal may be sent to animage histogram calculation block 615. Based on an analysis of the imagehistogram, e.g., an Adaptive AE strategy may be determined at block 620,which will apply an EV bias (either positive or negative) to the imagedata in conjunction with the color shading and vignetting corrected datavia a global DRC curve (625). The resulting tone mapped image is thenoutput as the image output signal.

Referring now to FIG. 6B, an adaptive AE process with decoupled colorshading and vignetting correction is shown in block diagram form 650,according to one embodiment. First, the camera focus position and scenelux level may be sent to vignetting modulation block 605.Simultaneously, the image input signal may be sent to an image histogramcalculation block 615. Based on an analysis of the image histogram,e.g., an Adaptive AE strategy may be determined at block 620, which willapply an EV bias (either positive or negative) to the image data inconjunction with the vignetting correction gains at block 640. Decoupledfrom the vignetting modulation process, color shading correction may beperformed at block 630. The output of the color shading correction andthe block 640 may then be tone mapped via the application of a pluralityof spatially-varying local DRC curves (635). The resulting tone mappedimage is then output as the image output signal.

Lux-Dependent and Focus-Dependent Vignetting Modulation

In one embodiment described herein, lens shading correction (LSC) may bemodulated based on scene lux level and focus distance. FIG. 7illustrates typical LSC gain curves 705 in graph form 700. Commonly, theLSC gains—once plotted in a 2D plot—exhibit a bowl-shapedcharacteristic. These gains compensate for both color lens shading(i.e., the deviation between R, G, and B channels) and vignetting. Thesolid lines 710 represent per-pixel normalized gains, as will beexplained in further detail below.

In low-light situations, the image noise might degrade the imagequality—especially at off-axis image positions since the shadingcorrection gains, g_(vig)(x), may become large in this image area(compare with the dashed-line curve 705 in FIG. 7). In the extreme caseof complete darkness, the sensor output is purely noise. In such a case,the bowl shaped shading gains do nothing but amplify noise. This canlead to unacceptable noise levels in the image. As scene illuminanceincreases, the sensor captures more signal from the actual scene, andthe lens shading effects begin to appear. To deal with the situation,the lens shading gains are configured to adaptively ‘scale down’ whenscene lux approaches zero and ‘scale up’ when scene lux changes fromnear zero to become larger. In other words, the lens shading correctionbehavior may be driven, at least in part, by sensor noise levels. Bycharacterizing scene noise levels, the system may avoid over-boostingnoise in the captured images.

Thus, in one embodiment described here, the lens shading gains aremodulated. For simplicity, a single function, g_(vig)(x), may be used torepresent the complete shading gain, encompassing both color shadinggain and vignetting gain. An exemplary adaptation scheme may beexpressed using the following formula, wherein g_(vig)(x) is blendedlinearly with an “identity” gain curve (i.e., gain of 1.0) driven by theAE digital gain or the estimated scene lux:

g′ _(vig)(x)=(1−z ₀)g _(vig)(x)+z ₀,  (Eqn. 6)

where z_(o0) is the modulation strength that, for instance, is assigneda value ranging from 1.0 to 0.25 at illuminance levels ranging from 50lux to 0 lux, respectively.

FIG. 8A illustrates an exemplary modulation of shading gains in the caseof low light scene conditions, whereas FIG. 8B illustrates an exemplarymodulation of shading gains driven by focus position. Curves 805/855represent the shading gains with a modulation value of 0.5; curves810/860 represent the shading gains with a modulation value of 0 (i.e.,macro lens setting in the case of FIG. 8B); and curves 815/865 representthe shading gains with a modulation value of 1.0 (i.e., infinity focusin the case of FIG. 8B).

As alluded to above with respect to FIG. 8B, in another embodiment ofthe lens shading gain modulation, the lens shading gain may be modulatedbased on the focus distance. For optical systems without zoom, theinventors have discovered that the lens shading fall off changes asfocus distance changes. The shorter focus distances have more cornershading fall off as the effective image system field of view increases.Signal landing on “off center” positions of the imaging sensor comesfrom a wider incoming angle (see, e.g., FIG. 8B, where the horizontalaxis represents a normalized diagonal field position from the center ofthe lens). Because different focus positions may lead to differentamounts of lens vignetting, it may be desirable to change g_(vig) basedon the lens focus position:

g″ _(vig)(x)=(1−z ₁)g _(vig) ^(macro)(x)+z ₁ g _(vig) ^(inf)(x),  (Eqn.7)

where 0<z₁<1 is the blending coefficient between a first vignettingcorrection function g_(vig) ^(macro)(x), which is obtained fromcalibration at macro lens settings and a second vignetting correctionfunction g_(vig) ^(inf)(x), which is obtained from calibration atinfinity focus. Modulating z₁ with the lens focus position may serve toimprove the vignetting correction by lowering the risk of over- orunder-correction across the focus range. The exact modulation mappingbetween z₁ and the lens position is derived from lens calibration dataover a full focus sweep. In other words, the lens shading correctionbehavior may also be driven, at least in part, by lens focus positions.By characterizing the optical system of the camera, the system may avoidover- or under-correcting for lens shading fall off changes. The scenelux-based modulation and focus position-based modulation may beimplemented alone or in conjunction with one another.

As illustrated in FIG. 6B, the total shading gain may be decoupled intocolor shading gains that are different between the different sensorcolor channels and a common vignetting gain that is same for all sensorcolor channels. Color shading gains may be used to compensate for thesensor pixel position-dependent light absorbing discrepancies betweendifferent color channels. Vignetting gains may be used to compensate forthe incoming light intensity fall off when the incoming light hits theimaging system at wider angle. The lens shading modulation techniquesdescribed above may be deployed on total lens shading gain curves of allsensor color channels, or may be deployed on only vignetting gaincurves. Alternately, the total shading gain may comprise luminanceshading only (i.e., rather than using different color shading gains foreach of the different sensor color channels), such that the chromashading remains untouched.

The lens shading gains may be normalized such that the minimum gain fromall color channels is always 1.0 at every pixel location. The gaincurves are illustrated as solid lines 705 in FIG. 7, whereas the dashedlines 710 represent the compound lens shading gains with color andvignetting correction characteristics. The normalized lens shading gainscorrect for the color lens shading portion, whereas the vignettingportion remains uncorrected. Later, the vignetting portion may becorrected using spatially-adaptive DRC curves in a Local Dynamic RangeCompensation (LDRC) process. These curves may be calculated according toEqns. 3 and 4 above, with the exception that the EV gain is replaced bya compound gain, g_(tot), which encompasses both the EV bias gain andthe spatially-varying vignetting gains g_(vig), g′_(vig), or g″_(vig),i.e., g_(tot)=g_(EV)*g_(vig), for instance. The linear segment of thefunction f_(DRC)(n) as used in Eqns. 3 and 4 may then be defined as

${f_{lin}(n)} = {\frac{n}{N}{g_{tot}.}}$

This decoupling of color shading gains and vignetting gains has severaladvantages. Firstly, the dynamic range is increased at off-axis pixellocations since the vignetting may cause more highlights to be capturedwithout the signal clipping. At the LDRC stage, the linear “up gaining”with g_(tot) will ensure that the vignetting is corrected for the low-and mid-tones, and the non-linear part of the curve will compress thehighlights. Secondly, the color shading is always fully corrected at theLSC stage, whereas the vignetting portion may be only partiallycorrected according to the lux- and focus-dependent modulation asdescribed earlier. Thirdly, the computation in the vignetting modulationneeds only to be calculated for one gain instead of four pixel gains(e.g., R, Gb, Gr, B) in the case that the color lens shading is carriedout on a Bayer color filter array raw image prior to RGB interpolation.

In the LDRC process, the Tone Mapping Operator, as described withreference to FIG. 2 above, may be replaced with a spatially-varying,i.e., Local DRC hue-preserving RGB Tone Mapping Curve. First, luminancevalues may be calculated from the RGBs for each pixel in the image.Next, gains are computed for local DRC tone mapping curves, and thegains for each local DRC tone mapping curve are then applied to theappropriate input RGB values to generate the desired output RGB levels.For example, in some embodiments, a local DRC tone mapping curve may bedefined for each pixel location in the image. In other embodiments,local DRC tone mapping curves may be defined for a plurality oflocations within the image, and local DRC tone mapping curves forindividual pixels may be determined by, e.g., interpolating between aplurality of the closest defined local DRC tone mapping curves.

Referring now to FIG. 9, a simplified functional block diagram of anillustrative electronic device 900 is shown according to one embodiment.Electronic device 900 may include processor 905, display 910, userinterface 915, graphics hardware 920, device sensors 925 (e.g.,proximity sensor/ambient light sensor, accelerometer and/or gyroscope),microphone 930, audio codec(s) 935, speaker(s) 940, communicationscircuitry 945, digital image capture unit 950, video codec(s) 955,memory 960, storage 965, and communications bus 970. Electronic device900 may be, for example, a personal digital assistant (PDA), personalmusic player, mobile telephone, or a notebook, laptop or tablet computersystem.

Processor 905 may be any suitable programmable control device capable ofexecuting instructions necessary to carry out or control the operationof the many functions performed by device 900 (e.g., such as thegeneration and/or processing of images in accordance with operations inany one or more of the Figures). Processor 905 may, for instance, drivedisplay 910 and receive user input from user interface 915 which cantake a variety of forms, such as a button, keypad, dial, a click wheel,keyboard, display screen and/or a touch screen. Processor 905 may be asystem-on-chip such as those found in mobile devices and include adedicated graphics processing unit (GPU). Processor 905 may be based onreduced instruction-set computer (RISC) or complex instruction-setcomputer (CISC) architectures or any other suitable architecture and mayinclude one or more processing cores. Graphics hardware 920 may bespecial purpose computational hardware for processing graphics and/orassisting processor 905 process graphics information. In one embodiment,graphics hardware 920 may include a programmable graphics processingunit (GPU).

Sensor and camera circuitry 950 may capture still and video images thatmay be processed to generate wide angle-of-view images, at least inpart, by video codec(s) 955 and/or processor 905 and/or graphicshardware 920, and/or a dedicated image processing unit incorporatedwithin circuitry 950. Images so captured may be stored in memory 960and/or storage 965. Memory 960 may include one or more different typesof media used by processor 905, graphics hardware 920, and image capturecircuitry 950 to perform device functions. For example, memory 960 mayinclude memory cache, read-only memory (ROM), and/or random accessmemory (RAM). Storage 965 may store media (e.g., audio, image and videofiles), computer program instructions or software, preferenceinformation, device profile information, and any other suitable data.Storage 965 may include one more non-transitory storage mediumsincluding, for example, magnetic disks (fixed, floppy, and removable)and tape, optical media such as CD-ROMs and digital video disks (DVDs),and semiconductor memory devices such as Electrically ProgrammableRead-Only Memory (EPROM), and Electrically Erasable ProgrammableRead-Only Memory (EEPROM). Memory 960 and storage 965 may be used toretain computer program instructions or code organized into one or moremodules and written in any desired computer programming language. Whenexecuted by, for example, processor 905 such computer program code mayimplement one or more of the methods described herein.

It is to be understood that the above description is intended to beillustrative, and not restrictive. The material has been presented toenable any person skilled in the art to make and use the invention asclaimed and is provided in the context of particular embodiments,variations of which will be readily apparent to those skilled in the art(e.g., some of the disclosed embodiments may be used in combination witheach other). In addition, it will be understood that some of theoperations identified herein may be performed in different orders. Thescope of the invention therefore should be determined with reference tothe appended claims, along with the full scope of equivalents to whichsuch claims are entitled. In the appended claims, the terms “including”and “in which” are used as the plain-English equivalents of therespective terms “comprising” and “wherein.”

1. A non-transitory program storage device, readable by a programmablecontrol device and comprising instructions stored thereon to cause theprogrammable control device to: obtain a first auto exposure (AE) targetfor a first image of a scene to be captured by an image sensor; obtainthe first image captured based, at least in part, on the first AEtarget, wherein the first image comprises a first plurality of pixels;determine whether there is unused dynamic range in the captured firstimage; determine a second AE target different from the first AE targetif it is determined that there is unused dynamic range; obtain a secondimage captured based, at least in part, on the second AE target, whereinthe second image comprises a second plurality of pixels; generate one ormore Dynamic Range Compensation (DRC) curves for the second image based,at least in part, on the first AE target; and perform tone mapping onthe second plurality of pixels using the generated one or more DRCcurves, wherein the tone mapping performed on the second plurality ofpixels is configured to render the second image to match a renderingluminance as if the second image was captured with the first AE target.2. The non-transitory program storage device of claim 1, wherein the oneor more DRC curves comprise a global DRC curve.
 3. The non-transitoryprogram storage device of claim 1, wherein the one or more DRC curvescomprise a plurality of spatially-varying local DRC curves.
 4. Thenon-transitory program storage device of claim 1, wherein theinstructions to perform tone mapping on the second plurality of pixelsfurther comprise instructions to digitally gain down the low and middletones of the second plurality of pixels.
 5. The non-transitory programstorage device of claim 1, wherein the instructions to perform tonemapping on the second plurality of pixels further comprise instructionsto expand the contrast of a highlight portion of the second plurality ofpixels.
 6. The non-transitory program storage device of claim 1, whereinthe instructions to perform tone mapping on the second plurality ofpixels further comprise instructions to ensure that white pixels withinthe second plurality of pixels remain white after the performance of thetone mapping.
 7. The non-transitory program storage device of claim 1,wherein the instructions to generate one or more DRC curves furthercomprise instructions to generate at least one DRC using a piece-wiselinear function.
 8. The non-transitory program storage device of claim1, wherein the instructions to determine the second AE target are based,at least in part, on an amount of motion blur in the first image.
 9. Anon-transitory program storage device, readable by a programmablecontrol device and comprising instructions stored thereon to cause theprogrammable control device to: obtain a first auto exposure (AE) targetfor a first image of a scene to be captured by an image sensor; obtainthe first image captured based, at least in part, on the first AEtarget, wherein the first image comprises a first plurality of pixels;determine whether any of the first plurality of pixels comprise clippedhighlight pixels; determine a second AE target different from the firstAE target if it is determined that clipped highlight pixels are present;obtain a second image captured based, at least in part, on the second AEtarget, wherein the second image comprises a second plurality of pixels;generate one or more Dynamic Range Compensation (DRC) curves for thesecond image based, at least in part, on the first AE target; andperform tone mapping on the second plurality of pixels using thegenerated one or more DRC curves, wherein the tone mapping performed onthe second plurality of pixels is configured to render the second imageto match a rendering luminance as if the second image was captured withthe first AE target.
 10. The non-transitory program storage device ofclaim 9, wherein the one or more DRC curves comprise a global DRC curve.11. The non-transitory program storage device of claim 9, wherein theone or more DRC curves comprise a plurality of spatially-varying localDRC curves.
 12. The non-transitory program storage device of claim 9,wherein the instructions to perform tone mapping on the second pluralityof pixels further comprise instructions to digitally gain up the low andmiddle tones of the second plurality of pixels.
 13. The non-transitoryprogram storage device of claim 9, wherein the instructions to performtone mapping on the second plurality of pixels further compriseinstructions to compress a highlight portion of the second plurality ofpixels.
 14. The non-transitory program storage device of claim 9,wherein the instructions to perform tone mapping on the second pluralityof pixels further comprise instructions to ensure that signal clippingdoes not occur after the performance of the tone mapping.
 15. Thenon-transitory program storage device of claim 9, wherein theinstructions to generate one or more DRC curves further compriseinstructions to generate at least one DRC using a piece-wise linearfunction.
 16. The non-transitory program storage device of claim 9,wherein the instructions to determine the second AE target are based, atleast in part, on an amount of noise in the first image.
 17. A system,comprising: an image capture device; a memory operatively coupled to theimage capture device and having, stored therein, computer program code;and a programmable control device operatively coupled to the memory andcomprising instructions stored thereon to cause the programmable controldevice to execute the computer program code to: obtain a first autoexposure (AE) target for a first image of a scene to be captured by theimage capture device; obtain the first image captured based, at least inpart, on the first AE target, wherein the first image comprises a firstplurality of pixels; determine whether any of the first plurality ofpixels comprise clipped highlight pixels or whether there is any unuseddynamic range in the captured first image; determine a second AE targetdifferent from the first AE target if it is determined that clippedhighlight pixels are present or if it is determined that there is anyunused dynamic range in the captured first image; obtain a second imagecaptured based, at least in part, on the second AE target, wherein thesecond image comprises a second plurality of pixels; generate one ormore Dynamic Range Compensation (DRC) curves for the second image based,at least in part, on the first AE target; and perform tone mapping onthe second plurality of pixels using the generated one or more DRCcurves, wherein the tone mapping performed on the second plurality ofpixels is configured to render the second image to match a renderingluminance as if the second image was captured with the first AE target.18. The system of claim 17, wherein the one or more DRC curves comprisea global DRC curve.
 19. The system of claim 17, wherein the one or moreDRC curves comprise a plurality of spatially-varying local DRC curves.20. The system of claim 17, wherein the instructions to generate one ormore DRC curves further comprise instructions to generate at least oneDRC using a piece-wise linear function.